chaos: vthread-context-cascade antagonist + glibc heap-corruption defaults#642
chaos: vthread-context-cascade antagonist + glibc heap-corruption defaults#642jbachorik wants to merge 5 commits into
Conversation
CI Test ResultsRun: #29490747291 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-07-16 10:45:05 UTC |
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/124973094 |
This comment has been minimized.
This comment has been minimized.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e1dd0fb1c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Benchmark Results (commit 3e379a7)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123961967 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
|
Not a review, but one thing that I can't quickly see from looking at the code: does it trigger actual migrations of vthreads across carriers? Because that is the fault scenario that led to the heap corruption. |
|
@rkennke It is trying to - but it is very probabilistic. It is trying to hold back carrier threads so new ones have to be spawned, then let the old one die so the vthread will have to be remounted but DBB still point to now invalid memory. |
J9
|
I see. If you have the time/nerve(/AI tokens) for it, you could perhaps try using a custom FJP where you can control the thread lifecycles. Probabilistic is fine, though, better than nothing. |
|
@rkennke That's a great idea. Thanks! |
Benchmark Results (commit b71ca62)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124279353 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10191 ms (21 iters) | ✅ 10399 ms (21 iters) | ≈ +2% (±11.9%) | — / — |
| akka-uct | 25 | ✅ 8940 ms (24 iters) | ✅ 8819 ms (24 iters) | ≈ -1.4% (±9.6%) | — / — |
| finagle-chirper | 21 | ✅ 5958 ms (33 iters) | ✅ 5999 ms (33 iters) | ≈ +0.7% (±24.8%) | |
| finagle-chirper | 25 | ✅ 5524 ms (36 iters) | ✅ 5491 ms (36 iters) | ≈ -0.6% (±25%) | |
| fj-kmeans | 21 | ✅ 2746 ms (68 iters) | ✅ 2742 ms (68 iters) | ≈ -0.1% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2737 ms (69 iters) | ✅ 2836 ms (66 iters) | 🔴 +3.6% | — / — |
| future-genetic | 21 | ✅ 2083 ms (89 iters) | ✅ 2065 ms (90 iters) | ≈ -0.9% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2041 ms (90 iters) | ✅ 2141 ms (87 iters) | 🔴 +4.9% | — / — |
| naive-bayes | 21 | ✅ 1291 ms (133 iters) | ✅ 1255 ms (136 iters) | ≈ -2.8% (±31.7%) | — / — |
| naive-bayes | 25 | ✅ 983 ms (174 iters) | ✅ 999 ms (171 iters) | ≈ +1.6% (±31.9%) | — / — |
| reactors | 21 | ✅ 16505 ms (15 iters) | ✅ 16271 ms (15 iters) | ≈ -1.4% (±8.3%) | — / — |
| reactors | 25 | ✅ 18606 ms (14 iters) | ✅ 18677 ms (15 iters) | ≈ +0.4% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2094 / 2059 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 5 | 2307 / 2237 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8584 / 8425 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 8769 / 8940 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1252 / 1221 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1320 / 1251 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 2917 / 2984 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2900 / 2964 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 3507 / 3542 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 8 / 2 | 3509 / 3455 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1682 / 1599 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1520 / 1920 | ✅ / ✅ | ✅ / ✅ |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b71ca62c0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Confirmed case 2. Traced the skip's history: the commit that added it ( |
Benchmark Results (commit 1d054cc)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124288733 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10249 ms (21 iters) | ✅ 10378 ms (21 iters) | ≈ +1.3% (±10.7%) | — / — |
| akka-uct | 25 | ✅ 8766 ms (24 iters) | ✅ 8849 ms (24 iters) | ≈ +0.9% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 5973 ms (33 iters) | ✅ 5999 ms (33 iters) | ≈ +0.4% (±25.1%) | |
| finagle-chirper | 25 | ✅ 5498 ms (36 iters) | ✅ 5448 ms (36 iters) | ≈ -0.9% (±24.6%) | |
| fj-kmeans | 21 | ✅ 2740 ms (68 iters) | ✅ 2777 ms (67 iters) | ≈ +1.4% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2813 ms (66 iters) | ✅ 2802 ms (66 iters) | ≈ -0.4% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2042 ms (90 iters) | ✅ 2218 ms (84 iters) | 🔴 +8.6% | — / — |
| future-genetic | 25 | ✅ 2098 ms (88 iters) | ✅ 2010 ms (93 iters) | 🟢 -4.2% | — / — |
| naive-bayes | 21 | ✅ 1269 ms (135 iters) | ✅ 1279 ms (134 iters) | ≈ +0.8% (±32.9%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1007 ms (170 iters) | ≈ -0.5% (±31.3%) | — / — |
| reactors | 25 | ✅ 18623 ms (15 iters) | ✅ 18109 ms (15 iters) | ≈ -2.8% (±5.7%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2035 / 2016 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2000 / 2280 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8876 / 8515 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8312 / 8747 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1277 / 1255 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 1287 / 1258 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2993 / 3089 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2898 / 2898 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 6 | 3554 / 3472 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 3475 / 3493 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1903 / 1926 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit fb6a1e8)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124310879 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10197 ms (21 iters) | ✅ 10305 ms (21 iters) | ≈ +1.1% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8925 ms (24 iters) | ✅ 8774 ms (24 iters) | ≈ -1.7% (±10.1%) | — / — |
| finagle-chirper | 21 | ✅ 5925 ms (33 iters) | ✅ 5973 ms (33 iters) | ≈ +0.8% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5436 ms (36 iters) | ✅ 5402 ms (36 iters) | ≈ -0.6% (±23.7%) | |
| fj-kmeans | 21 | ✅ 2651 ms (71 iters) | ✅ 2759 ms (68 iters) | 🔴 +4.1% | — / — |
| fj-kmeans | 25 | ✅ 2837 ms (66 iters) | ✅ 2843 ms (66 iters) | ≈ +0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2038 ms (91 iters) | ✅ 2134 ms (87 iters) | 🔴 +4.7% | — / — |
| future-genetic | 25 | ✅ 2085 ms (89 iters) | ✅ 2107 ms (88 iters) | ≈ +1.1% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1234 ms (138 iters) | ✅ 1249 ms (136 iters) | ≈ +1.2% (±32.6%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1023 ms (167 iters) | ≈ +0.7% (±31.6%) | — / — |
| reactors | 21 | ✅ 16230 ms (15 iters) | ✅ 16350 ms (15 iters) | ≈ +0.7% (±8.4%) | — / — |
| reactors | 25 | ✅ 18749 ms (15 iters) | ✅ 18774 ms (15 iters) | ≈ +0.1% (±5%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2022 / 1981 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2389 / 2312 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 8854 / 8395 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 8242 / 8401 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 8 | 1286 / 1277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 1293 / 1264 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2929 / 2992 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 2895 / 2906 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 3507 / 3474 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 3485 / 3437 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1966 / 1937 | ✅ / ✅ | ✅ / ✅ |
|
Root-caused the reliability crashes reported in Confirmed the crash requires the ddprof agent (the same antagonist run with no The harness was never passing Worth calling out: |
|
@copilot resolve the merge conflicts in this pull request |
Benchmark Results (commit c7bf65c)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124374293 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10216 ms (21 iters) | ✅ 10186 ms (21 iters) | ≈ -0.3% (±11.2%) | — / — |
| akka-uct | 25 | ✅ 8851 ms (24 iters) | ✅ 8864 ms (24 iters) | ≈ +0.1% (±9.9%) | — / — |
| finagle-chirper | 21 | ✅ 6035 ms (33 iters) | ✅ 5965 ms (33 iters) | ≈ -1.2% (±24.8%) | |
| finagle-chirper | 25 | ✅ 5451 ms (36 iters) | ✅ 5420 ms (36 iters) | ≈ -0.6% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2791 ms (67 iters) | ✅ 2784 ms (67 iters) | ≈ -0.3% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2835 ms (66 iters) | ✅ 2822 ms (66 iters) | ≈ -0.5% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2112 ms (88 iters) | ✅ 2119 ms (88 iters) | ≈ +0.3% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2091 ms (89 iters) | ✅ 1947 ms (95 iters) | 🟢 -6.9% | — / — |
| naive-bayes | 21 | ✅ 1325 ms (129 iters) | ✅ 1239 ms (137 iters) | ≈ -6.5% (±31.7%) | — / — |
| naive-bayes | 25 | ✅ 1023 ms (167 iters) | ✅ 1005 ms (170 iters) | ≈ -1.8% (±31.4%) | — / — |
| reactors | 21 | ✅ 16789 ms (15 iters) | ✅ 15597 ms (16 iters) | 🟢 -7.1% | — / — |
| reactors | 25 | ✅ 18388 ms (15 iters) | ✅ 19200 ms (14 iters) | ≈ +4.4% (±4.7%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2004 / 1877 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 2212 / 2130 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 9142 / 8336 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8043 / 8491 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1278 / 1245 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 1289 / 1272 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2963 / 2969 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2990 / 2839 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 3537 / 3505 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 6 / 3 | 3488 / 3463 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1779 / 1596 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1851 / 2013 | ✅ / ✅ | ✅ / ✅ |
Resolved by merging |
Benchmark Results (commit 2e34507)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124382794 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Benchmark Results (commit cfe1245)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124478924 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Benchmark Results (commit 33a8f3d)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124498331 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10359 ms (21 iters) | ✅ 10432 ms (21 iters) | ≈ +0.7% (±11%) | — / — |
| akka-uct | 25 | ✅ 8965 ms (24 iters) | ✅ 8729 ms (24 iters) | ≈ -2.6% (±10.8%) | — / — |
| finagle-chirper | 21 | ✅ 6009 ms (33 iters) | ✅ 6032 ms (33 iters) | ≈ +0.4% (±25.1%) | |
| finagle-chirper | 25 | ✅ 5446 ms (36 iters) | ✅ 5464 ms (36 iters) | ≈ +0.3% (±24.1%) | |
| fj-kmeans | 21 | ✅ 2702 ms (69 iters) | ✅ 2788 ms (67 iters) | 🔴 +3.2% | — / — |
| fj-kmeans | 25 | ✅ 2748 ms (68 iters) | ✅ 2757 ms (67 iters) | ≈ +0.3% (±2.8%) | — / — |
| future-genetic | 21 | ✅ 2153 ms (86 iters) | ✅ 2092 ms (89 iters) | 🟢 -2.8% | — / — |
| future-genetic | 25 | ✅ 2057 ms (91 iters) | ✅ 2038 ms (91 iters) | ≈ -0.9% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1257 ms (136 iters) | ✅ 1280 ms (134 iters) | ≈ +1.8% (±32.7%) | — / — |
| naive-bayes | 25 | ✅ 1026 ms (167 iters) | ✅ 982 ms (173 iters) | ≈ -4.3% (±30.9%) | — / — |
| reactors | 21 | ✅ 17155 ms (15 iters) | ✅ 16050 ms (15 iters) | ≈ -6.4% (±7.4%) | — / — |
| reactors | 25 | ✅ 18361 ms (15 iters) | ✅ 18840 ms (15 iters) | ≈ +2.6% (±3.7%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2050 / 1918 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 2273 / 2115 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8651 / 8737 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 8617 / 8305 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 1262 / 1289 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1279 / 1244 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2954 / 3068 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2901 / 2857 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 3474 / 3522 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 7 / 3 | 3505 / 3513 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1780 / 1682 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1893 / 1910 | ✅ / ✅ | ✅ / ✅ |
|
@copilot resolve the merge conflicts in this pull request |
Resolved by merging |
Benchmark Results (commit 17d0792)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124513178 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10378 ms (21 iters) | ✅ 10302 ms (21 iters) | ≈ -0.7% (±11.7%) | — / — |
| akka-uct | 25 | ✅ 8861 ms (24 iters) | ✅ 8870 ms (24 iters) | ≈ +0.1% (±9.7%) | — / — |
| finagle-chirper | 21 | ✅ 6046 ms (33 iters) | ✅ 6109 ms (33 iters) | ≈ +1% (±25.7%) | |
| finagle-chirper | 25 | ✅ 5515 ms (36 iters) | ✅ 5589 ms (35 iters) | ≈ +1.3% (±25.3%) | |
| fj-kmeans | 21 | ✅ 2780 ms (67 iters) | ✅ 2656 ms (70 iters) | 🟢 -4.5% | — / — |
| fj-kmeans | 25 | ✅ 2831 ms (66 iters) | ✅ 2820 ms (66 iters) | ≈ -0.4% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2067 ms (89 iters) | ✅ 2164 ms (86 iters) | 🔴 +4.7% | — / — |
| naive-bayes | 21 | ✅ 1283 ms (133 iters) | ✅ 1289 ms (133 iters) | ≈ +0.5% (±33.1%) | — / — |
| naive-bayes | 25 | ✅ 968 ms (175 iters) | ✅ 1008 ms (170 iters) | ≈ +4.1% (±32.5%) | — / — |
| reactors | 21 | ✅ 15934 ms (15 iters) | ✅ 16648 ms (15 iters) | ≈ +4.5% (±8.1%) | — / — |
| reactors | 25 | ✅ 18828 ms (15 iters) | ✅ 18359 ms (15 iters) | ≈ -2.5% (±4.8%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2003 / 2011 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2162 / 2344 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8597 / 8674 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8651 / 8404 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 1284 / 1291 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1283 / 1277 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2971 / 3043 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 2874 / 2900 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 3512 / 3521 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 3440 / 3533 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1649 / 1644 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1911 / 1819 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit be3c3bf)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124562849 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10226 ms (21 iters) | ✅ 10174 ms (21 iters) | ≈ -0.5% (±10.9%) | — / — |
| akka-uct | 25 | ✅ 8753 ms (24 iters) | ✅ 8941 ms (24 iters) | ≈ +2.1% (±10.1%) | — / — |
| finagle-chirper | 21 | ✅ 5920 ms (33 iters) | ✅ 6013 ms (33 iters) | ≈ +1.6% (±25.1%) | |
| finagle-chirper | 25 | ✅ 5449 ms (36 iters) | ✅ 5493 ms (36 iters) | ≈ +0.8% (±25.2%) | |
| fj-kmeans | 21 | ✅ 2761 ms (68 iters) | ✅ 2778 ms (67 iters) | ≈ +0.6% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2743 ms (68 iters) | ✅ 2822 ms (66 iters) | 🔴 +2.9% | — / — |
| future-genetic | 21 | ✅ 2057 ms (91 iters) | ✅ 2122 ms (87 iters) | 🔴 +3.2% | — / — |
| future-genetic | 25 | ✅ 2099 ms (88 iters) | ✅ 2105 ms (88 iters) | ≈ +0.3% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1276 ms (134 iters) | ✅ 1252 ms (137 iters) | ≈ -1.9% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1014 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.3% (±31.5%) | — / — |
| reactors | 21 | ✅ 15190 ms (15 iters) | ✅ 16624 ms (15 iters) | 🔴 +9.4% | — / — |
| reactors | 25 | ✅ 18330 ms (15 iters) | ✅ 18900 ms (15 iters) | ≈ +3.1% (±4.4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2013 / 1963 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 7 | 2311 / 2165 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8339 / 8974 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8490 / 8255 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 1255 / 1304 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1272 / 1254 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2940 / 2992 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2880 / 2882 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 9 | 3513 / 3481 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1457 / 1670 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1843 / 1882 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit ae3daa1)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124604420 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10249 ms (21 iters) | ✅ 10262 ms (21 iters) | ≈ +0.1% (±10.8%) | — / — |
| akka-uct | 25 | ✅ 8927 ms (24 iters) | ✅ 9068 ms (24 iters) | ≈ +1.6% (±10.9%) | — / — |
| finagle-chirper | 21 | ✅ 5962 ms (33 iters) | ✅ 6010 ms (33 iters) | ≈ +0.8% (±25.6%) | |
| finagle-chirper | 25 | ✅ 5482 ms (36 iters) | ✅ 5470 ms (36 iters) | ≈ -0.2% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2712 ms (69 iters) | ✅ 2721 ms (69 iters) | ≈ +0.3% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2829 ms (66 iters) | ✅ 2855 ms (66 iters) | ≈ +0.9% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2049 ms (90 iters) | ✅ 2126 ms (88 iters) | 🔴 +3.8% | — / — |
| future-genetic | 25 | ✅ 2047 ms (91 iters) | ✅ 2024 ms (92 iters) | ≈ -1.1% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1266 ms (135 iters) | ✅ 1352 ms (127 iters) | ≈ +6.8% (±33.9%) | — / — |
| naive-bayes | 25 | ✅ 1013 ms (169 iters) | ✅ 1026 ms (166 iters) | ≈ +1.3% (±32%) | — / — |
| reactors | 21 | ✅ 16418 ms (15 iters) | ✅ 15608 ms (15 iters) | ≈ -4.9% (±8.2%) | — / — |
| reactors | 25 | ✅ 18164 ms (15 iters) | ✅ 17971 ms (15 iters) | ≈ -1.1% (±6.4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1939 / 2026 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 2164 / 2347 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 8291 / 8431 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 8283 / 8356 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1278 / 1291 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1274 / 1286 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2934 / 3045 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2890 / 2824 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 7 | 3502 / 3552 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 3464 / 3429 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1765 / 1536 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1844 / 1973 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit 2a3e926)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124688617 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Benchmark Results (commit d1f2a0f)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124747414 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10284 ms (21 iters) | ✅ 10144 ms (21 iters) | ≈ -1.4% (±11.2%) | — / — |
| akka-uct | 25 | ✅ 8813 ms (24 iters) | ✅ 8820 ms (24 iters) | ≈ +0.1% (±9.7%) | — / — |
| finagle-chirper | 21 | ✅ 6022 ms (33 iters) | ✅ 6007 ms (33 iters) | ≈ -0.2% (±25.4%) | |
| finagle-chirper | 25 | ✅ 5472 ms (36 iters) | ✅ 5464 ms (36 iters) | ≈ -0.1% (±24.7%) | |
| fj-kmeans | 21 | ✅ 2765 ms (68 iters) | ✅ 2829 ms (66 iters) | ≈ +2.3% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2804 ms (66 iters) | ✅ 2813 ms (66 iters) | ≈ +0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2040 ms (91 iters) | ✅ 2152 ms (86 iters) | 🔴 +5.5% | — / — |
| future-genetic | 25 | ✅ 2029 ms (92 iters) | ✅ 2091 ms (89 iters) | 🔴 +3.1% | — / — |
| naive-bayes | 21 | ✅ 1270 ms (136 iters) | ✅ 1270 ms (134 iters) | ≈ 0% (±32.5%) | — / — |
| naive-bayes | 25 | ✅ 1019 ms (168 iters) | ✅ 1029 ms (166 iters) | ≈ +1% (±31.7%) | — / — |
| reactors | 21 | ✅ 16566 ms (15 iters) | ✅ 16611 ms (15 iters) | ≈ +0.3% (±7.3%) | — / — |
| reactors | 25 | ✅ 18343 ms (15 iters) | ✅ 18406 ms (15 iters) | ≈ +0.3% (±6.8%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2015 / 1901 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 2301 / 2165 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 8670 / 8927 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 8793 / 8343 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 1265 / 1259 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 7 / ✅ | 1282 / 1256 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2964 / 2930 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2932 / 2896 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 6 | 3525 / 3504 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 3474 / 3487 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1770 / 1625 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1745 / 1812 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit 2ef0fd2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124764606 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
kaahos
left a comment
kaahos
left a comment
There was a problem hiding this comment.
I've started to take a look at it and left some comments!
Benchmark Results (commit 08390ce)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124799195 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10375 ms (21 iters) | ✅ 10149 ms (21 iters) | ≈ -2.2% (±10.8%) | — / — |
| akka-uct | 25 | ✅ 8799 ms (24 iters) | ✅ 8810 ms (24 iters) | ≈ +0.1% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 6004 ms (33 iters) | ✅ 5987 ms (33 iters) | ≈ -0.3% (±24.9%) | |
| finagle-chirper | 25 | ✅ 5479 ms (36 iters) | ✅ 5475 ms (36 iters) | ≈ -0.1% (±24.1%) | |
| fj-kmeans | 21 | ✅ 2749 ms (68 iters) | ✅ 2820 ms (66 iters) | ≈ +2.6% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2871 ms (66 iters) | ✅ 2849 ms (66 iters) | ≈ -0.8% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2027 ms (92 iters) | ✅ 2133 ms (88 iters) | 🔴 +5.2% | — / — |
| future-genetic | 25 | ✅ 2031 ms (91 iters) | ✅ 2110 ms (88 iters) | 🔴 +3.9% | — / — |
| naive-bayes | 21 | ✅ 1299 ms (131 iters) | ✅ 1319 ms (131 iters) | ≈ +1.5% (±32.8%) | — / — |
| naive-bayes | 25 | ✅ 1010 ms (170 iters) | ✅ 1014 ms (169 iters) | ≈ +0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 16954 ms (15 iters) | ✅ 16040 ms (15 iters) | ≈ -5.4% (±7.8%) | — / — |
| reactors | 25 | ✅ 18876 ms (15 iters) | ✅ 18543 ms (15 iters) | ≈ -1.8% (±3.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1961 / 2029 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2142 / 2180 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 8351 / 8465 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 8501 / 8600 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1273 / 1282 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1308 / 1277 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 3025 / 2980 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2846 / 2882 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 3479 / 3552 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 3476 / 3468 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1819 / 1741 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1852 / 1863 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit 68ceb93)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124817306 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10309 ms (21 iters) | ✅ 10255 ms (21 iters) | ≈ -0.5% (±11.3%) | — / — |
| akka-uct | 25 | ✅ 8778 ms (24 iters) | ✅ 8848 ms (24 iters) | ≈ +0.8% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 6033 ms (33 iters) | ✅ 5964 ms (33 iters) | ≈ -1.1% (±25.4%) | |
| finagle-chirper | 25 | ✅ 5465 ms (36 iters) | ✅ 5488 ms (36 iters) | ≈ +0.4% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2779 ms (67 iters) | ✅ 2736 ms (68 iters) | ≈ -1.5% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2839 ms (66 iters) | ✅ 2842 ms (66 iters) | ≈ +0.1% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2039 ms (90 iters) | ✅ 2072 ms (89 iters) | ≈ +1.6% (±2.5%) | — / — |
| future-genetic | 25 | ✅ 2069 ms (89 iters) | ✅ 2002 ms (92 iters) | 🟢 -3.2% | — / — |
| naive-bayes | 21 | ✅ 1245 ms (138 iters) | ✅ 1288 ms (133 iters) | ≈ +3.5% (±33.1%) | — / — |
| naive-bayes | 25 | ✅ 1022 ms (167 iters) | ✅ 1028 ms (166 iters) | ≈ +0.6% (±32.1%) | — / — |
| reactors | 21 | ✅ 16345 ms (15 iters) | ✅ 16388 ms (15 iters) | ≈ +0.3% (±8.8%) | — / — |
| reactors | 25 | ✅ 18285 ms (15 iters) | ✅ 18580 ms (15 iters) | ≈ +1.6% (±4.4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 2006 / 2091 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2306 / 2458 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 8510 / 8550 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8164 / 8361 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 1265 / 1270 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2929 / 2905 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 2873 / 2848 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 2 | 3570 / 3504 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 10 / 5 | 3485 / 3462 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1722 / 1701 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1897 / 1886 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit 8c1402d)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124958407 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10448 ms (21 iters) | ✅ 10477 ms (21 iters) | ≈ +0.3% (±11.6%) | — / — |
| akka-uct | 25 | ✅ 8898 ms (24 iters) | ✅ 8852 ms (24 iters) | ≈ -0.5% (±10%) | — / — |
| finagle-chirper | 21 | ✅ 6046 ms (33 iters) | ✅ 5993 ms (33 iters) | ≈ -0.9% (±25.1%) | |
| finagle-chirper | 25 | ✅ 5523 ms (36 iters) | ✅ 5495 ms (36 iters) | ≈ -0.5% (±25.6%) | |
| fj-kmeans | 21 | ✅ 2786 ms (68 iters) | ✅ 2717 ms (68 iters) | ≈ -2.5% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2774 ms (67 iters) | ✅ 2827 ms (66 iters) | ≈ +1.9% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2140 ms (87 iters) | ✅ 2057 ms (90 iters) | 🟢 -3.9% | — / — |
| naive-bayes | 21 | ✅ 1248 ms (137 iters) | ✅ 1259 ms (136 iters) | ≈ +0.9% (±32.7%) | — / — |
| naive-bayes | 25 | ✅ 994 ms (172 iters) | ✅ 990 ms (172 iters) | ≈ -0.4% (±31.9%) | — / — |
| reactors | 21 | ✅ 15822 ms (15 iters) | ✅ 15857 ms (15 iters) | ≈ +0.2% (±8.1%) | — / — |
| reactors | 25 | ✅ 18626 ms (15 iters) | ✅ 18484 ms (15 iters) | ≈ -0.8% (±4.2%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1960 / 1971 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 4 | 2212 / 2311 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 9050 / 8435 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8780 / 8664 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1290 / 1255 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1293 / 1252 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2925 / 2912 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 3517 / 3529 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 3473 / 3496 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1529 / 1671 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1863 / 1884 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit 9ef1d89)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124973098 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10293 ms (21 iters) | ✅ 10304 ms (21 iters) | ≈ +0.1% (±11%) | — / — |
| akka-uct | 25 | ✅ 8858 ms (24 iters) | ✅ 8998 ms (24 iters) | ≈ +1.6% (±10.7%) | — / — |
| finagle-chirper | 21 | ✅ 5996 ms (33 iters) | ✅ 5966 ms (33 iters) | ≈ -0.5% (±25.6%) | |
| finagle-chirper | 25 | ✅ 5557 ms (35 iters) | ✅ 5528 ms (36 iters) | ≈ -0.5% (±24.9%) | |
| fj-kmeans | 21 | ✅ 2748 ms (68 iters) | ✅ 2817 ms (66 iters) | ≈ +2.5% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2760 ms (68 iters) | ✅ 2841 ms (66 iters) | 🔴 +2.9% | — / — |
| future-genetic | 21 | ✅ 2089 ms (89 iters) | ✅ 2118 ms (87 iters) | ≈ +1.4% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2113 ms (88 iters) | ✅ 2043 ms (92 iters) | 🟢 -3.3% | — / — |
| naive-bayes | 21 | ✅ 1247 ms (137 iters) | ✅ 1252 ms (136 iters) | ≈ +0.4% (±33%) | — / — |
| naive-bayes | 25 | ✅ 1009 ms (170 iters) | ✅ 1022 ms (167 iters) | ≈ +1.3% (±32.1%) | — / — |
| reactors | 21 | ✅ 16251 ms (15 iters) | ✅ 16951 ms (15 iters) | ≈ +4.3% (±7.5%) | — / — |
| reactors | 25 | ✅ 18539 ms (15 iters) | ✅ 18552 ms (15 iters) | ≈ +0.1% (±5.7%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 8 | 1960 / 1846 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2318 / 2258 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 8564 / 8577 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 8449 / 8280 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1246 / 1260 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1266 / 1291 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 3004 / 2918 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2906 / 2986 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 3509 / 3504 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 3509 / 3520 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1647 / 1844 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1869 / 1997 | ✅ / ✅ | ✅ / ✅ |
rkennke
left a comment
rkennke
left a comment
There was a problem hiding this comment.
Review
Overall this is careful, well-commented work — the antagonist and runner reflect a lot of hard-won operational knowledge (documented job IDs, hysteresis, keep-alive reasoning). My main concerns are scope and blast radius, not correctness of the core antagonist.
Key issues
1. Scope creep: the frame.h / frame_ut.cpp change is unrelated to this PR.
The isRawPointer gating change has nothing to do with the vthread antagonist or malloc defaults. The production change itself is correct — encode() always sets ENCODED_MASK (frame.h:63) and RAW_POINTER_MASK is only ever set alongside it, so requiring ENCODED_MASK before trusting bit 30 is sound, and the new regression test is good. But bundling a production behavior change into a "chaos tooling" PR hides it from reviewers who'd otherwise scrutinize the packing logic. Recommend splitting it into its own PR.
2. Two tests were deleted with no explanation (frame_ut.cpp): FrameTypeEncodeTest.EncodedValuesArePositive and FrameTypeDecodeTest.DecodedTypeIsInValidRange. Both look like they'd still pass and both cover behavior orthogonal to the isRawPointer change. Deleting passing tests silently reduces coverage — if they were redundant or wrong, please say so; otherwise keep them.
3. MALLOC_CHECK_/MALLOC_PERTURB_ blast radius in ProfilerTestPlugin.kt. This applies to every ddprof-test task, not just the chaos suite.
- Non-reproducible failures:
MALLOC_PERTURB_is(1..255).random()(and$((RANDOM % 255 + 1))in the shell) and the value is never logged. When a run aborts on corruption you can't reproduce it with the same perturb byte. Consider logging the chosen value (Gradle test output + the runner's stdout). - Surfacing latent issues suite-wide is presumably intended, but worth calling out in the PR body — this can turn previously-green unrelated tests red.
- Minor: the guard is
!testEnv.containsKey("LD_PRELOAD"). IfLD_PRELOADis set in the process environment rather than the task'stestEnvmap, the guard won't see it. Confirm the sanitizer/tcmalloc configs all populatetestEnv.
Smaller notes
MemoryGovernorSystem.gc()race: two threads can both observecritical == false, both set it, and both fireSystem.gc(). The comment promises a "one-shot nudge"; it's approximately-once, not once. Harmless, but the code is slightly weaker than the comment claims.paceCalls.incrementAndGet()on every hot-loop iteration (MemoryGovernor) is a single contendedAtomicLongshared across all antagonist threads — a cache-line hotspot that perturbs the very allocation timing you're stressing. A per-thread counter would avoid it if it ever matters.fastCarrierSchedulerisThreadPoolExecutor(0, Integer.MAX_VALUE, …, SynchronousQueue)— unbounded platform threads, bounded only indirectly byMAX_STALE_RACERS = 1024. Worth confirming ~1024 OS carrier threads stay under the container thread/RSS ceilings on aarch64.start()hard-fails on a VT-capable JDK without--add-opens(throws, crashing the harness). Intentional and documented, andrun-chaos-harness.shpasses the flag — but it's a tight coupling: any alternate invocation path that forgets the flag hard-crashes at startup. Acceptable given it's test-only.run-chaos-harness.shfind … -name a -o -name brelies on the implicit-printattaching to the whole OR expression. It works, but wrapping in\( … \)makes intent explicit and robust against a future added clause.stopGracefullyjoins the three driver threads unconditionally. Safe here because astart()that threw is never registered inMain's running list, but a null-guard or comment would prevent a shutdown NPE from masking the real failure.
Correctness / tests / security
- Correctness: the one production change (
frame.h) is verified correct against the bit layout; semaphore acquire/release pairing in both antagonists correctly releases on the reflective-failure path (no permit leaks). - Test coverage: the new
isRawPointerregression test is good; the two unexplained deletions are the gap (#2). - Conventions: comment style, copyright headers, and Jira-only-in-PR-body all respected.
- Security: test/CI tooling only; the
dtdg.codownload and Maven snapshot fallback are unchanged from the prior script.
Bottom line
The chaos antagonist and runner rework are solid. Before merge I'd (1) split out the frame.h change, (2) restore or justify the two deleted frame_ut.cpp tests, and (3) log the MALLOC_PERTURB_ value so glibc-detected aborts are reproducible. The rest is optional polish.
|
Ouch, the |
A raw, unencoded ASGCT BCI can coincidentally have bit 30 set on non-HotSpot VMs; isRawPointer() must require ENCODED_MASK before trusting bit 30, not just the HotSpot check.
Races tracer-style context propagate/activate/restore semantics against virtual-thread carrier churn, targeting the ContextStorageMode.THREAD stale-carrier DirectByteBuffer use-after-free.
…onist MemoryGovernor watches heap/direct-memory usage and gates DirectMemoryAntagonist.burstLoop() plus the other allocation-heavy antagonists on it, nudging GC once on a critical watermark and exiting cleanly on an early heap OOME while tolerating a late one. VirtualThreadChurnAntagonist bounds in-flight virtual threads with a semaphore: the default carrier pool has far fewer platform threads than a batch submits, so an unthrottled submission rate outpaces execution and the backlog of pending VirtualThread/Continuation objects grows without bound.
…rruption detection utils/run-chaos-harness.sh is the new standalone entry point (usable from CI or by hand); chaos_check.sh becomes a thin wrapper around it. Also defaults MALLOC_CHECK_/MALLOC_PERTURB_ for glibc-backed test runs (this script and the shared ddprof-test Gradle test-task configuration), turning silent heap corruption into an immediate, attributable SIGABRT instead of a crash much later in an unrelated allocation. Skipped on musl and whenever a sanitizer/allocator already replaces malloc via LD_PRELOAD. The chosen MALLOC_PERTURB_ byte is logged so a corruption abort can be reproduced.
…nded fast-carrier pool MemoryGovernor: throttled/critical are now AtomicBoolean with compareAndSet-guarded transitions, so a watermark crossing observed by several antagonist threads at once escalates (and fires System.gc()) exactly once instead of racing. paceCalls moves from a shared AtomicLong to a per-thread counter, removing a cache-line contention point on every antagonist's hot allocation loop. VirtualThreadContextCascadeAntagonist: fastCarrierScheduler's max pool size is now bounded by MAX_STALE_RACERS instead of Integer.MAX_VALUE (no headroom lost, since that's already the hard cap on concurrent racers). stopGracefully null-checks its driver threads so it's safe to call even if start() never ran or failed partway through; same guard added to VirtualThreadChurnAntagonist. run-chaos-harness.sh: the allocator-detection find commands now cap -maxdepth 4 instead of unboundedly recursing /usr/lib, /usr/lib64, etc.
What does this PR do?:
Adds a new
vthread-context-cascadechaos antagonist that races tracer-stylecontext propagate/activate/restore semantics against virtual-thread carrier
churn, targeting the
ContextStorageMode.THREADstale-carrierDirectByteBufferuse-after-free. Also defaultsMALLOC_CHECK_/MALLOC_PERTURB_for glibc-backed test runs (chaos reliability runner andthe shared
ddprof-testGradle test-task configuration), turning silentheap corruption into an immediate, attributable
SIGABRTinstead of acrash much later in an unrelated allocation.
Motivation:
While building the
vthread-context-cascadereproducer it became clearthat this class of use-after-free bug corrupts glibc heap/tcache metadata
silently and only manifests as a crash far later, in an unrelated
allocation call — making it unlikely to be caught within a short chaos
run or CI test window. Enabling glibc's own heap-corruption checks by
default closes that gap cheaply, without needing a sanitizer build (which
can't run against a live JVM, see
reference_sanitizers_no_jvm).Additional Notes:
The
MALLOC_CHECK_/MALLOC_PERTURB_defaults are skipped on musl (noMALLOC_CHECK_support) and whenever a sanitizer/allocator alreadyreplaces malloc via
LD_PRELOAD(ASan/TSan/tcmalloc/jemalloc configs),mirroring the existing
gmallocvstcmalloc/jemallocmutualexclusivity in
chaos_check.sh.How to test the change?:
./gradlew :ddprof-stresstest:chaosJarbuilds cleanly with the new antagonist registered../gradlew :build-logic:conventions:compileKotlincompiles cleanly with theProfilerTestPluginchange.-Dddprof.debug.context.storage.mode=thread, tuned virtual-thread scheduler flags,MALLOC_CHECK_=3/MALLOC_PERTURB_set) for several minutes; no crash reproduced yet in this window — the antagonist is being landed as a reliability-suite addition to keep exercising this race over longer soak runs, not as a guaranteed-repro smoke test.For Datadog employees:
If this PR touches code that signs or publishes builds or packages, or handles
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.This PR doesn't touch any of that.
JIRA: PROF-15360
Unsure? Have a question? Request a review!